Introduction

The credibility of scientific research is in doubt, among lay consumer (Hornsey & Fielding, 2017) and scientist (Pashler & Wagenmakers, 2021) alike. Several tools have been proposed to combat this “crisis of confidence” (Ibid., p. 528). One such tool is the crowd science approach: “the organization of scientific research in open and collaborative projects” (Franzoni & Sauermann, 2014, p. 1). We focus on crowdsourced data analysis, also known as the many analysts or multi-analyst approach: giving the same dataset to different teams of scientists, who independently analyze it to answer the same research question and/or estimate a parameter of interest.
\(~~~~~\) According to science reformers, crowd-scientific findings that tell a consistent story should garner more confidence in the conclusions and increase public faith in science (Silberzahn et al., 2018; Uhlmann et al., 2019). Here, we ask if we can find empirical evidence for these claims: Does crowdsourcing data analysis improve lay perceptions of scientific findings?

Objectives

We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay consumers’ posterior beliefs, perceptions of credibility, confidence in an aggregate effect size estimate, and ratings of researcher bias, error, and discretion.
\(~~~~~\) We compare the effects of providing lay consumers with a single, aggregate parameter estimate (the single-analyst condition) vs. multiple parameter estimates that (a) vary slightly and are all positive, leading to the same qualitative conclusion (the multi-consistent condition) or (b) vary widely and are of both signs, leading to differing qualitative conclusions (the multi-inconsistent condition). In all three conditions, the given estimates average to 5%.

Preregistered Hypotheses

Table 1: Predicted direction of effects

Note. Table 1 indicates the predicted direction of effects for all outcomes, compared to the single-analyst condition and controlling for prior beliefs (a green plus/red minus indicates a positive/negative prediction, respectively). For example, we hypothesized that, compared to a single-analyst study and controlling for prior beliefs, ratings of credibility would be greater in the multi-consistent condition and lower in the multi-inconsistent condition.

Methods

We run an experiment (N = 1,498) with three conditions
\(~\) Single-analyst
A single, aggregate parameter estimate
\(~\) Multi-consistent
Multiple parameter estimates with low variance and high consensus
\(~\) Multi-inconsistent
Multiple parameter estimates with high variance and low consensus

Experimental Design

Results

Figure 1: Estimates relative to the single-analyst condition In line with our hypotheses, lay consumers of multi-analyst studies with inconsistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Find the results less credible
\(~~~\) Have less confidence in the average effect size estimate
\(~~~\) Believe the results are more likely to stem from bias
\(~~~\) Believe the results are more likely to stem from error

Contrary to our hypotheses, lay consumers of multi-analyst studies with consistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Believe the results are more likely to stem from error

We found no significant effects on
\(~~~\) Credibility of the results
\(~~~\) Confidence in the effect size estimate
\(~~~\) Ratings of bias

Exploratory results

For the additional exploratory measure, lay consumers of multi-analyst studies (both with consistent and inconsistent results)
\(~~~\) Perceive greater researcher degrees of freedom

Figure 2: Distribution of prior and posterior beliefs by condition

Discussion

Conclusion
\(~\) Crowdsourced data analysis has many worthy uses, but…
\(~~\) It’s highly resource-intensive
\(~\) Variability and lack of consensus may evoke negative responses \(~~\) Surprisingly, even multiple, consistent estimates may backfire

Future Directions
\(~~~\) Perceptions of scientists?
\(~~\) Science communication and communicating uncertainty

Open Science Statement
The preregistration, survey materials, data, and code that support the findings of this study are openly available on GitHub and the OSF.
\(~~\) [Insert GitHub link here]
\(~~\) [Insert OSF link here]